Distributed optimization of deeply nested systems

•Many architectures share the fundamental design principle of constructing a deeply nested mapping from inputs to outputs. •Learning these architectures is challenging because nesting (i.e., function composition) produces inherently nonconvex functions. •Backprop suffers from vanishing gradients and is hard to parallelize, is only applicable if the mappings are differentiable with respect to the parameters, and needs careful tuning of learning rates. •Selecting the best architecture, for example the number of units in each layer of a deep net, or the number of filterbanks in a speech front-end processing, requires a combinatorial search. •We describe a general optimization strategy called method of auxiliary coordinates (MAC). It has provable convergence, is easy to implement reusing existing algorithms for single layers, can be parallelized trivially and massively, applies even when parameter derivatives are not available or not desirable, can perform some model selection on the fly, and is competitive with state-ofthe-art nonlinear optimizers even in the serial computation setting, often providing reasonable models within a few iterations. x y z1 z2 z3 W1 W2 W3 W4 σ